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ENHANCED DISPLAY OF WORLD WIDE WEB PAGES ON TELEVISION 
Field of the invention 

The present invention relates to the combined display of 
television signals and World Wide Web pages. 

Background of the invention 

It has been proposed to combine various forms of media. In 
particular, there are proposed systems to combine the World 
Wide Web (WWW) portion of the Internet and television video. 
Television video is typically available by terrestrial 
broadcast, cable and satellite. In addition, other methods of 
television delivery are known. 

The World Wide Web is most commonly available through an 
Internet service provider over a dial up telephone modem on a 
plain telephone line. More recently other forms of high speed 
Internet access (ISDN, ASDL or broadband cable modem and the 
like) are available. However, while many devices incorporate 
access to both television media and the Internet, there is 
still a problem of how to merge the two media (television and 
WWW Internet) in such manner that the merged media is both 
useful to the viewer and intuitive to use. 
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In some cases, a single television screen is shared by 
switching between television viewing and Internet surfing. For 
example, a television show may contain a reference to a Web 
site. The viewer switches the single screen from television 
viewing to the referenced Web site. After navigating to the 
referenced Web site and from there to possibly other portions 
of the World Wide Web, the viewer switches back to watching 
television. In the alternative, a personal computer monitor 
used for Internet surfing may also be used for television video 
viewing . 

In other cases, a single screen is shared between simultaneous 
television viewing and Internet surfing. One method of 
simultaneous screen sharing is to split the screen such that 
television viewing is provided in a first portion of the screen 
and a Web page is viewed in a second portion of the screen. In 
a special case of screen sharing, the well-known "picture-in- 
picture" (PIP) format is provided in which a smaller size image 
is placed within a larger (normal size) background image. 
Although the smaller PIP image covers up a portion of the 
larger (full size, full screen) background image, the viewer 
may be able to avoid covering up an important part of the 
larger background image by selecting the position and size of 
the smaller PIP image. 

When applied to combining WWW and television video using PIP 
techniques, the television video image is typically the smaller 



image and World Wide Web graphics is the background image. The 
PIP format is sometimes referred to as PIG for u picture-in- 
graphics" . Both picture-in-graphics and the reverse, graphics- 
in-picture are included in the term picture-in-picture, or PIP. 

In the case of a hypertext markup language (HTML) Web page 
(graphics) as the background image, and television video as the 
PIG image, the Web page may be scrolled, while the television 
video PIG image remains stationary. Thus, if the television 
video PIG image covers up an important part of the larger HTML 
background image, the viewer must scroll the HTML background 
image out from under the television video PIG image or move the 
PIG image. In general, prior art PIP image displays do not 
allow simultaneous display of all of two images, TV and WWW 
graphics, at the same time. 

Summary of the invention 

In accordance with the present invention, a system for 
enhancing the display of World Wide Web pages combined with 
television video on a video screen is provided. In particular, 
the viewer is provided with control over transparency of a PIP 
image. In such manner a PIP image, which may normally cover up 
an important part of the background Web page image, is made 
transparent so that the user can view the background Web page 
image through the transparent PIP image. With transparency 
control, two images having the same size may be simultaneously 
viewed. A full size background image and a full size foreground 
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image may simultaneously occupy the full television video 
screen . 

In accordance with another aspect of the present invention, a 
television video PIG image is embedded in an HTML Web page as 
an object. When the HTML Web page is displayed as a background 
image and scrolled (or panned) , the television video PIG image 
scrolls along with the HTML Web page background image. In such 
manner, the smaller PIG image has its own space (or box) on the 
Web page. The smaller PIG image moves with the Web graphics 
image and does not cover up an important part of the background 
Web graphics image. In the latter embedded mode, the PIG image 
is not just an overlay on top of the web page, but instead is 
an integral part of the web page. 

Brief description of the drawings 

Figure 1 is a block diagram of a CATV system embodying the 
present invention . 

Figures 2A shows a prior art display illustrating an HTML Web 
page with an opaque television video PIG image on top. 

Figures 2B and 2C show the prior display of figure 2A in which 
the HTML Web page in the background is scrolled up while the 
PIG image in the foreground remains at its original position 
relative to the display screen. 



1176 

• • 

Figure 3A illustrates an HTML Web page with a transparent 
television video PIG image on top, in accordance with the 
present invention . 

Figures 3B and 3C show the display of figure 3A in which the 
HTML Web page is scrolled up and down while the PIG image in 
the foreground remains at its original position relative to 
the display screen, in accordance with the present invention. 

Figure 4A is an illustration of an HTML Web page with a 
television video PIG image embedded in the HTML Web page, in 
accordance with the present invention. 

Figure 4B shows the display of figure 4A in which the Web page 
is scrolled up with the PIG image being an integral part of 
the HTML Web page, in accordance with the present invention. 

Figures 5a-5e illustrate various transparency modes for 
picture-in-graphics (PIG) television video overlay on an HTML 
Web page in accordance with the present invention. 

Figure 6 is a block diagram of the display logic for embedding 
a television video image within an HTML Web page in accordance 
with the present invention. 
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Figure 7 is a flow diagram illustrating the different stages 
of processing for embedding a television video image within an 
HTML Web page in accordance with the present invention. 

Figure 8 is a program listing in pseudo code illustrating an 
HTML extension used for embedding a television image object in 
a Web page . 

Detailed description 

A television video and data display system for use with the 
present invention is shown in figure 1. A distribution network 
100 broadcasts television video and World Wide Web data to a 
plurality of settop boxes 101 for display on individual 
television sets 102. The distribution network 100 may be any 
suitable broadband medium such as wired coaxial cable or fiber 
optic cable or a wireless broadband medium such as direct 
satellite broadcast or terrestrial transmission. The settop 
box 101 provides video 104 and audio 105, 106 signals to the 
television display 102 . 

The settop box 101 stores a software module (the client 
software) downloaded from the distribution network 100. The 
client software runs locally on the settop box 101 and 
performs the following functions. 

1) Obtains World Wide Web content and television programming 
content from the distribution network 100. 



1176 

2) Interprets the relationship of the World Wide Web content 
to the television video. 

3) Generates a composite display of integrated television 
video and graphics in the settop box 101 for display on the 
television 102 . 

In the prior art, it is known to combine a television video 
image with a Web page by a Picture-In-Picture overlay in the 
foreground with a World Wide Web page as the background. The 
behavior of the combined PIP/background image is illustrated 
in figures 2A-2C. In figure 2A, an opaque television video 
image 604 is in the foreground overlaid on Web page 600, 
partially obscuring the underlying Web page 600. As the Web 
page is scrolled upward 601 in figure 2B the opaque television 
video image 604 obscures a different portion of the underlying 
Web page background. As the viewer continues to scroll the Web 
page upward 602 in figure 2C the opaque television video image 
604 obscures yet a different portion of the underlying Web 
page background. Thus, in order to view all of the Web page 
600 while having a television video image in a Picture-In- 
Picture overlay 604, the viewer must scroll the background Web 
page vertically 601, 602 (or pan horizontally) out from under 
the overlaid television video image 604. 

Figures 3A-3C illustrate a display in accordance with the 
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present invention. In figure 3A, the viewer has control over 
the transparency of the television video image 503. 
Transparent (or translucent) television video image 503 is in 
the foreground, overlaid on Web page 500. By being 
transparent, television video image 503 does not obscure Web 
page 500. As the Web page is scrolled upward 501 in figure 3B, 
the transparent television video image 503 permits the user to 
view a different portion of the Web page 500 without obscuring 
the portion of the Web page underlying the television video 
image. As the user further scrolls the Web page upward 5 02 in 
figure 3C, the transparent television video image 503 always 
permits the user to view the portion of the Web page 
background underneath the television video image 503. Thus, 
the user can view the Web page through the television video 
image in a Picture-In-Picture overlay, either while the Web 
page is stationary or while the viewer scrolls the Web page 
past the transparent television video image. 

An alternative to the use of transparency to prevent an 
overlaid television video image from obscuring an HTML web 
page, is shown in figures 4A and 4B. That is, figures 4 A and 
4B illustrate a display in accordance with an alternate 
embodiment of the present invention. In figure 4A, a 
television video image 7 02, 7 03 is embedded as an object in a 
Web page 7 00. When the Web page is scrolled upward 701 in 
figure 4B, the embedded television video image 703 also 
scrolls upward. By placing (embedding) the television video 
image 7 02, 7 03 as an object (and therefore in an unused 
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portion) of the Web page 700 701, the television video image 
702 does not obscure the Web page 700, 701. 

Figures 5A-5E show different modes of operation based on 
viewer control over the size, position and degree of 
transparency (or level of translucency) of the Picture-In- 
Graphics (PIG) television video image. In figures 5A-5E Web 
page 400 occupies the full television screen with an overlaid 
PIG image window (402, 404, 406, 408, 410) . In figure 5A, the 
PIG image window 402 is overlaid on Web page 400. The Web page 
region 400 outside of PIG image 402 is opaque (100% 
weighting) . The PIG window 402 is a digital television video 
image, which is made translucent with the Web page. 
Specifically, each of the pixels in television video image 402 
are weighted 2 5% from video and 7 5% from the Web page, which 
results in a very transparent television video image 402 on 
the underlying Web page 400. 

The translucency values are selectable, and any value may be 
used for the weighting factor. For example, in figure 5B, each 
pixel in television video image 404 is weighted 50% from video 
and 50% from the Web page, which results in a moderate 
transparency in which the television video image 402 and the 
Web page 400 are equally weighted. In figure 5C, each pixel in 
television video image 406 is weighted 75% video and 25% Web 
page, which results in a slightly transparent television video 
image 402 on the underlying Web page 400. If, as in figure 5D, 
each pixel in television video image 408 is weighted 100% 
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video and 0% Web page, the television video image 4 08 PIG 
IMAGE becomes opaque. 

Background and foreground are relative terms with respect to 
transparency. If the video and Web page pixels are weighted 
75/25 as in figure 5C, the video is in the foreground and the 
Web page is in the background. If the video and Web page are 
weighted 2 5/75 as in figure 5A, the video is in the background 
and the Web page is in the foreground. If the video and Web 
page pixels are equally weighted 50/50, neither is background 
or foreground. As transparency values for the video and the 
Web page are changed from 5 0/50, one becomes background and 
the other becomes foreground. 

The PIG image window may also occupy the full screen, allowing 
mixing of a full screen television video image 410 with a full 
screen Web page 400, as shown in figure 5E. In figure 5E, the 
pixels in the full screen television video image 410 are 
weighted 50% video and 50% Web page. The full screen 
television video image 410 is also used with the different 
levels of translucency between the television video 410 and 
the Web page 400. 

A block diagram of a display generator for embedding 
television video within a Web page is shown in figure 6. The 
inputs to the display generator are MPEG audio and video 203, 
HTML web data 202 and user control inputs 201. The output of 
the display generator is the composite video and audio output 
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213 to the television display. 

The display generator comprises a parsing and layout engine 
206 coupled to a screen management module 207 which is further 
coupled to a video and graphics (GFX) control engine 208. A 
graphic memory 209 stores the rendered graphics output from 
the screen management module 2 07 corresponding to the HTML 2 02 
input graphics and text. 

A video processor 210 is provided to decode the received MPEG 
input signal 203. The output of the video processor 210 is 
coupled to a video resizing module 211. A display formatter 
212 is responsive to the video resizing module 211, the 
graphics memory 2 09 and the resizing and transparency controls 
from the Video/GFX control engine 208. 

In operation, the parsing and layout engine 2 06 conditions the 
HTML data for display on a television screen and extracts the 
specific video related information from the HTML source 202. 
The output of the parsing and layout engine 2 06 is fed to the 
screen management module 2 07 that renders the Web page in the 
graphics memory 209. The screen management module 207 forwards 
video integration parameters to the Video/GFX graphics) 
control engine 208. 

The Video/GFX control engine 2 08 takes the user inputs 2 01, 
which consist of Picture-In-Graphics (PIG) controls 205 and 
transparency controls 204. PIG image controls 205 determine 
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desired size and position for the inserted Picture-In- 
Graphics. Transparency controls 204 relate to the degree of 
transparency for the inserted Picture-In-Graphics . The PIG 
image controls 2 05 and transparency controls 2 04 control the 
Display Formatter 212 and the video resizing engine 211 in 
accordance with the video integration parameters 214 that were 
extracted from the incoming HTML source 2 02 by the screen 
management module 2 07. 

The MPEG Audio/Video compressed stream 203 is routed to the 
video processor 210 for decoding. The Video Resizing module 
211 is responsible to resize the video in real time (on the 
fly) . The display formatter module 212 and the video resizing 
module 211 allow flexible combinations of graphics data 215 
and real time video stream data 216 to form a various Picture 
In Graphics (PIG) and transparency combinations at the output 
213, as illustrated in figures 3A-3C, 4A, 4B and 5A-5E. 

The display behavior and integration of Web content and 
television video is controlled by the incoming HTML 2 02 
source. With the present system, the author of a Web page 
using a markup language like HTML can integrate television 
video with Web based text and graphics together to create a 
compelling user interface on a television screen. The markup 
language tag for the video (800 in figure 8) controls the 
display position, style and behavior of the video embedded 
within the Web page on the settop box. 
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An HTML scheme to specify television video and Web data 
integration is illustrated in figures 7 and 8. In figure 7, 
the settop box downloads a markup language page 3 08 from the 
network and processes it to determine the position, size and 
other attributes of the television video along with the 
display attributes for text and graphics in the Web page 
content. The whole processing of the markup language page 3 08 
is done in software in various modules illustrated in figure 
7. 

Following steps comprise the process: 

Step 1: The Network protocol module 300 of the software 
downloads the Markup Language file 3 08 through a network 
connection from a remote host. 

Step 2 : The information downloaded from the network is passed 
to the parser module 3 01 which parses the markup language 
based on a predetermined syntax and grammar and separates all 
the markup tags, attributes associated with the tags and the 
data information. Based on the tags parsed from the page, the 
parser 301 builds a logical structure for the document 304 and 
all the entities 305 (see 801 in figure 8) that consists of 
the document 304. The television video, which is treated as an 
entity, is also constructed based on the video tag as 
specified in the markup language. For illustration, the syntax 
for specifying the video entity in the markup language is as 
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f ollows : 



<VIDEO SRC = " Source: Frequency: ChannelNo" HEIGHT= " in graphical 
units" WIDTH=" in graphical units" BORDER= " in graphical units "> 



SRC - specifies the source for the video. The source for the 
video could specified by the frequency to which the settop box 
tuner needs to be tuned and/or the user perceived channel 
number that the user selects. 

HEIGHT - height of the video region to be displayed. 
WIDTH - width of the video region to be displayed. 
BORDER - border around the video region. 



<PIG X="X position on the display absolute coordinate" or X="X 
position in % of display width" Y=" Y position on the display 
absolute coordinate " or Y ="Y position in % of display 
height" RATIO or WIDTH="" HEIGHT=""> 

</PIGPOS> 



Where 



<HEAD> 



<PIGPOS> 



</HEAD> 
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The document 304 thus constructed at end of the parsing stage 
3 01 is then passed to the next stage of processing for 
calculation of all the physical attributes required to display 
the document on television monitor. 

Step 3: The structured document /entity representation of the 
page is next processed by the layout module 3 02 that decides 
the position, look and feel of each entity 305 within the Web 
page which comprise of either television video, text or 
graphics. The position information and other display 
attributes such as color, border etc. for each entity are 
represented in a logical structure called a box 306 (see 802 
in figure 8) . The box created contains all the information, 
necessary and required by the render/display module 303, to 
render the entity on the screen. The document now has a list 
of boxes 306 including a video box for the television video, 
that has the dimensions and display attributes according to 
the intent of the content author. Each box has a box type 
associated data field to distinguish it as being a box for 
text, a box for graphics or a box for television video. 

Step 4: The render and display module 303 takes the list of 
boxes 3 06 within the document 3 04A as its input and renders 
all the boxes, one by one, into the display buffer (209 in 
figure 6) . The render/display module 303 in figure 7 
encompasses the functions of screen management 2 07 and 
video/GFX control 208 in figure 6. When the display module 
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encounters a video type box in the document, the display 
module (208 in figure 6) switches to the appropriate real time 
video source. The television video 310 is merged on the fly 
with the Web graphics and text 312 for an integrated look and 
feel. 

Several modes of operation are supported: 

MODE 1 - Full screen transparent Web page with full 
screen transparent television video. 

Full screen mode is shown in figure 5E. The Web page 400 
occupies the entire resolution of the television screen and is 
transparent allowing the viewer to see the full screen 
television video 410. The transparency values are shown as 50% 
for the Web page 400 and 50% for the television video 410. 
Since the Web page 400 and the television video 410 are of 
equal weight neither can be considered background or 
foreground. However, the transparency values for television 
video versus Web page are adjustable by the user. The viewer 
can fade either one so as to put the Web page in the 
foreground and the television video in the background, or vice 
versa . 

MODE 2 - Full screen Web page background with transparent 
television video in non-embedded overlaid Picture In 
Graphics (television video overlay which does not scroll 
with Web page background) 

Overlay mode with transparency control is shown in figures 3A, 



1176 

3B and 3C. The Web page 500 occupies the entire resolution of 
the television screen and is opaque. The broadcast television 
video is resized in a translucent ( PIG) Picture-In-Graphics 
box 503. (The PIG image may be opaque as in figures 2A-2C) . 
The PIG image is positioned as an overlay on top of the Web 
page and its position and size are independent of the Web page 
in the background. As shown in figures, scrolling the Web page 
500, 501 and 502 in figures 3A, 3B and 3C respectively does 
not change the PIG image 503 position or its size on the 
television screen . 

MODE 3 - Full screen Web page opaque with embedded 
television video Picture-In-Graphics (television video 
overlay which scrolls with Web page background) 

Embedded mode is shown in figures 4A and 4B. The Web page 7 00 
occupies the entire resolution of the television screen and is 
opaque. The broadcast television video is resized in a 
Picture-In-Graphics box 702, 703 and is embedded in the Web 
page 700, 701. The PIG image 702, 703 is integral part of the 
Web page like any other asset, such as text, images, tables 
and the like. When the Web page 7 00, 701 scrolls (vertical 
screen movement and/or pans (horizontal screen movement, the 
television video PIG image 702, 703 will move as well in order 
to keep its relative position in the Web page 700, 701. 



